How to Do Tie-breaking in Prioritization of Interaction Test Suites?

The prioritization of interaction test suites has received more attention in the field of combinatorial interaction testing, especially when testing resources are limited to allow the part of combinatorial test cases to be executed. Many strategies have been proposed to prioritize interaction test suites according to different evaluation measures. However, most of these strategies may face a challenge to choose more than one “best” candidate with the largest evaluation measure value. In this case, there is a tie among all “best” candidates. How to do tie-breaking? Intuitively speaking, random tiebreaking could be a reasonable choice, which has also been applied to many research papers. In this paper, we investigate different tie-breaking techniques including random tiebreaking, first-element tie-breaking, last-element tie-breaking, higher-strength tie-breaking, and lower-strength tie-breaking, and also conduct experiments on a well-known prioritization strategy of interaction test suites, namely interaction coverage based prioritization, in order to present a guideline of choosing tie-breaking techniques for testers in practical testing. The experimental results show that although no tie-breaking technique always performs best, in many cases random tie-breaking and last-element tie-breaking have best performance, so that they would be best choices for testers in the prioritization of interaction test suites. Keywords-Combinatorial interaction testing, interaction test suite, test case prioritization, tie-breaking, guideline